Recently, a startup in Seattle named Moondream launched a compact visual language model called moondream2. Despite its small size, the model has shown outstanding performance in various benchmark tests and has garnered significant attention. As an open-source model, moondream2 is expected to enable local image recognition capabilities on smartphones. Moondream2 was officially released in March and can process both text and image inputs, possessing functions such as answering questions, text extraction (OCR), object counting, and item classification.

AI startup Moondream has officially announced the completion of $4.5 million in seed funding and presents a disruptive viewpoint: in the world of AI models, smaller models may hold advantages. The company is backed by Felicis Ventures, Microsoft's M12 GitHub Fund, and Ascend, launching a visual language model with only 1.6 billion parameters that can compete with models four times its size in terms of performance.

Recently, H2O.ai announced the launch of two new visual language models aimed at enhancing the efficiency of document analysis and optical character recognition (OCR) tasks. The two models, H2OVL Mississippi-2B and H2OVL-Mississippi-0.8B, demonstrate remarkable competitiveness compared to models from large tech companies, potentially offering businesses dealing with heavy document workflows a more efficient solution.

Alibaba Group's Tongyi Qwen QwenLM project has encountered an unexpected takedown of its Github page, resulting in a 404 error message when users try to access it. Project leader Lin Junyang responded in a social media post, stating that the team has not disappeared and is in communication with officials to resolve the page takedown issue. Despite the team's efforts, the page remains inaccessible. It is worth noting that the team recently released the Qwen2-VL model, which has excelled in processing video content up to 20 minutes long, surpassing multiple authoritative evaluation metrics for multimodal models, with some metrics even exceeding those of GPT.

On September 2nd, Tongyi Qwen announced the open sourcing of its second-generation visual language model Qwen2-VL, and launched APIs for the 2B and 7B sizes as well as their quantized versions on the Aliyun Bailian platform for direct user access. The Qwen2-VL model achieves comprehensive performance improvements in several areas. It can understand images of different resolutions and aspect ratios, setting global leading performance on benchmarks such as DocVQA, RealWorldQA, and MTVQA.

NVIDIA has collaborated with several universities to introduce NVEagle, a large visual language model capable of chatting using images. NVEagle can analyze image content and provide accurate answers, such as identifying individuals in images, like Jensen Huang. The model significantly enhances the understanding of visual information by transforming images into visual tokens and combining them with text embeddings. In addressing the challenges of high-resolution image processing, the research team has constructed models like Eagle-X5-7B and Eagle-X by exploring various visual encoders and fusion strategies.

Recently, significant progress has been made in the research and application of multimodal large models. Foreign companies such as OpenAI, Google, and Microsoft have introduced a series of advanced models, and domestic institutions such as Zhipu AI and Jietu Xingchen have also made breakthroughs in this field. These models typically rely on visual encoders to extract visual features and integrate them with large language models, but they face issues such as visual inductive bias due to training

The Tsinghua University ZhiPu AI team has released a new visual language model called CogAgent, which focuses on understanding and navigating graphical user interfaces (GUIs). CogAgent uses a dual-encoder system to process complex GUI elements and text, showing outstanding performance with high-resolution inputs of 1120x1120 pixels. The model outperforms existing LLM methods in GUI navigation tasks on PC and Android platforms, and also excels in text and visual question-answering benchmarks. Potential applications include automated GUI operations, providing G

Zhipu AI has open-sourced CogAgent, a visual language model with 18 billion parameters. CogAgent excels in GUI understanding and navigation, achieving state-of-the-art general performance across multiple benchmark tests. The model supports high-resolution visual input and dialog Q&A, and can answer questions based on any GUI screenshot. CogAgent also supports OCR-related tasks, with its capabilities significantly enhanced through pre-training and fine-tuning.

Google has released the PaLI-3 visual language model, which is lightweight yet achieves SOTA performance. This model employs a contrastive pre-training method and deeply explores the potential of VIT, demonstrating outstanding performance in multilingual modality retrieval. PaLI-3 perfectly integrates natural language understanding and image recognition, becoming an important force in AI innovation. The model's SigLIP-based contrastive pre-training method opens up a new era for multilingual cross-modal retrieval. Although it has not been fully open-sourced, multilingual and English SigLIP models have been released, providing opportunities for researchers to experiment.

The newly proposed image reasoning benchmark HallusionBench is used to examine visual language models like GPT-4V, revealing issues with language and visual hallucinations. Models like GPT-4V exhibit a high error rate of up to 90% in generating language hallucinations influenced by parametric memory within HallusionBench. Additionally, models such as GPT-4V are prone to geometric and other visual illusions, indicating that their current visual capabilities are still limited. Simple image manipulations can easily mislead these models, reflecting their fragility.

Following the open sourcing of the general model Qwen-7B in August, Alibaba Cloud has released the visual language model Qwen-VL. Qwen-VL supports both Chinese and English and can perform tasks such as image captioning, image Q&A, and document Q&A. Qwen-VL also enables open-domain visual localization in Chinese by marking detection boxes within images.